Henrique Madeira (University of Coimbra)

The goal of dependability benchmarking is to provide generic ways for characterizing the behaviour of components and computer systems in the presence of faults, allowing for the quantification of dependability measures. Beyond existing evaluation techniques, dependability benchmarking must provide a reproducible and cost-effective way of performing such a characterization, especially for comparative purposes. This chapter presents the DBench framework for defining dependability benchmarks for computer systems, with particular emphasis on off-the-shelf (OTS), commercial (COTS) or not, and also OTS-based systems. The concepts and the multiple dimensions of the problem are discussed and the key aspects of the specification of dependability benchmarks are presented. A benchmark validation approach is proposed, consisting mainly on the verification of a set of well-defined properties that key benchmark components should meet in order to be meaningful under economically acceptable conditions. A tool for On-Line Analytical Processing (OLAP) of experiment results allowing analysing, sharing, and cross-exploiting results from dependability benchmarking experiments is then presented. The chapter ends with a short overview of the dependability benchmarks specified and implemented in DBench. DBench Dependability Benchmarking Concepts 1-2 1.